New go-runner image for distroless scenarios #90804

dims · 2020-05-06T13:22:07Z

Context : please see #90674

The Kubernetes go-runner image wraps the gcr.io/distroless/static image and provides a go based
binary that can run commands and wrap stdout/stderr etc.

Why do we need this? Some of our images like kube-apiserver currently use bash for collecting
logs, so we are not able to switch to distroless images directly for these images. The klog's
--log-file was supposed to fix this problem, but we ran into trouble in scalability CI jobs
around log rotation and picked this option instead. we essentially publish a multi-arch
manifest with support for various platforms. This can be used as a base for other kubernetes
components.

For example instead of running kube-apiserver like this:

"/bin/sh",
  "-c",
  "exec /usr/local/bin/kube-apiserver {{params}} --allow-privileged={{pillar['allow_privileged']}} 1>>/var/log/kube-apiserver.log 2>&1"

we would use go-runner like so:

"/go-runner", "--log-file=/var/log/kube-apiserver.log", "--also-stdout=false", "--redirect-stderr=true",
  "/usr/local/bin/kube-apiserver",
  "--allow-privileged={{pillar['allow_privileged']}}",
  {{params}}

The go-runner would then ensure that we run the /usr/local/bin/kube-apiserver with the
specified parameters and redirect stdout ONLY to the log file specified and ensure anything
logged to stderr also ends up in the log file.

Signed-off-by: Davanum Srinivas davanum@gmail.com

What type of PR is this?
/kind feature

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

dims · 2020-05-06T13:22:35Z

/priority important-soon
/sig testing
/sig release

dims · 2020-05-06T13:24:03Z

/assign @justaugustus
/assign @tallclair

dims · 2020-05-06T13:24:39Z

/assign @BenTheElder @fejta

dims · 2020-05-06T13:25:05Z

/sig testing
/sig release

dims · 2020-05-06T15:02:12Z

/retest

justaugustus · 2020-05-06T16:55:04Z

This is exciting!
/area release-eng security dependency

tallclair

I still don't quite understand why we can't just integrate this code directly into klog, but I guess if it unblocks the distroless transition I'm OK with it. I'd still like us to try and fix the klog issue though, and then move off of this.

build/go-runner/Dockerfile

build/go-runner/Makefile

build/go-runner/README.md

build/go-runner/go-runner.go

BenTheElder · 2020-05-07T01:28:49Z

I like this. distroless++

~~I guess we need to make sure callers setting the entrypoint on pods for this update, if we think that exists might be worth a release note of some sort.~~

EDIT: I guess no relnote for introducing the image, relnote possibly when we start using it?

Signed-off-by: Davanum Srinivas <davanum@gmail.com>

BenTheElder · 2020-05-07T01:45:07Z

/lgtm
/approve
/hold

BenTheElder · 2020-05-07T01:52:05Z

/hold cancel
edit above re: relnote.

dims · 2020-05-07T02:33:07Z

/test pull-kubernetes-e2e-kind

dims · 2020-05-07T03:21:52Z

/test pull-kubernetes-kubemark-e2e-gce-big

dims · 2020-05-07T08:57:47Z

/retest

tallclair · 2020-05-07T20:34:54Z

I like this. distroless++

BTW, I prototyped a tiny go utility that would dump stdin to an executable file. The usecase is to pre-load it into distroless containers for debugging (until we have debug containers) by doing:

$ cat busybox | kubectl exec -i <target_pod> -- /usr/bin/copier /busybox
$ kubectl exec <target_pod> -- /busybox --install /bin
$ kubectl exec <target_pod> -it /bin/sh
# ...

I'm not sure it's worth following up on, but it was a fun hack.

dims · 2020-05-08T01:15:24Z

Very cool @tallclair

BenTheElder · 2020-05-08T04:31:25Z

That is pretty sweet. Given that the images auto push now, any reason not to?

…

On Thu, May 7, 2020, 18:15 Davanum Srinivas ***@***.***> wrote: Very cool @tallclair <https://github.com/tallclair> — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#90804 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHADK5VCUW64IZ76H4QGOLRQNMLXANCNFSM4M2OCMIQ> .

This reverts commit 97278f1. Here is the sequence of events. Originally, kubernetes#849 incorrectly promoted sha256:536ab131b0d0e3b13eb83c985cc0ac9ba7e69e7dac9521e6cacd6a8b6019e0a6 to `v0.1.0`. This was in error because this digest was not the digest of the manifest list, but only for the amd64 image. Then, PR kubernetes#850 was created to fix this. Originally, that PR assigned the correct digest of the manifest list sha256:8ee20934e6c005a9ce8d6d8b7ed23698c3bb80e0b30a3d49e5aeca928cc69bf3 to a new tag, `v0.1.1`. This was OK for the promoter (it was what I LGTM'ed), because it assigned a new tag for a new digest; however it was NOT OK because the underlying image was really versioned as `v0.1.0` (kubernetes/kubernetes#90804). Later, PR kubernetes#850 *was changed* so that sha256:8ee20934e6c005a9ce8d6d8b7ed23698c3bb80e0b30a3d49e5aeca928cc69bf3 was tagged as `v0.1.0` while a new build of `v0.1.1` was created here kubernetes/kubernetes#90852. This change to promote sha256:8ee20934e6c005a9ce8d6d8b7ed23698c3bb80e0b30a3d49e5aeca928cc69bf3 into the __existing__ `v0.1.0` tag resullted in PR kubernetes#850 becoming a NOP. The promoter simply ignored this PR because tag moves are not supported. That is, in order to honor the intent of PR kubernetes#850, the promoter would have had to __delete__ the `v0.1.0` tag from the incorrect digest sha256:536ab131b0d0e3b13eb83c985cc0ac9ba7e69e7dac9521e6cacd6a8b6019e0a6 and reassign this tag to sha256:8ee20934e6c005a9ce8d6d8b7ed23698c3bb80e0b30a3d49e5aeca928cc69bf3. This is why the promoter complained about this in PR kubernetes#850's postsubmit run about tag moves, and is still complaining about this even after PR kubernetes#853 (promoting the newly-built `v0.1.1` image) was merged, like this: ``` ... tag 'v0.1.0' in dest points to sha256:536ab131b0d0e3b13eb83c985cc0ac9ba7e69e7dac9521e6cacd6a8b6019e0a6, not sha256:8ee20934e6c005a9ce8d6d8b7ed23698c3bb80e0b30a3d49e5aeca928cc69bf3 (as per the manifest), but tag moves are not supported; skipping ... ``` Our promoter manifests should be kept free of impossible-to-do intent. The solution is to either (1) delete the existing `v0.1.0` tag from production (making the intent not a tag move, but a tag add) and keep the promoter manifest as-is, or (2) revert PR kubernetes#850 and keep the incorrect digest for `v0.1.0` to silence the tag move warning. Because this is not a production emergency, (1) would be against policy. Hence this PR, which opts for option (2). That being said, the dry run of the promoter in PR kubernetes#850 did correctly detect the tag move as well, but exited without an error. We should exit with an error on tag moves during dry runs in the future, because tag moves result from prohibited manifest *intent*. This is tracked here: kubernetes-sigs/promo-tools#212.

k8s-ci-robot · 2022-08-22T17:17:55Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: BenTheElder, dims, Elimuhub-coder

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~build/OWNERS~~ [BenTheElder,dims]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels May 6, 2020

k8s-ci-robot requested review from lavalamp and spiffxp May 6, 2020 13:23

k8s-ci-robot assigned justaugustus and tallclair May 6, 2020

k8s-ci-robot assigned BenTheElder and fejta May 6, 2020

k8s-ci-robot added sig/testing Categorizes an issue or PR as relevant to SIG Testing. sig/release Categorizes an issue or PR as relevant to SIG Release. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels May 6, 2020

dims force-pushed the new-go-runner-image-for-distroless branch from 5628ea8 to 67f0a5f Compare May 6, 2020 13:30

dims mentioned this pull request May 6, 2020

Add new image-pushing job for go-runner kubernetes/test-infra#17530

Merged

k8s-ci-robot added area/release-eng Issues or PRs related to the Release Engineering subproject area/security area/dependency Issues or PRs related to dependency changes labels May 6, 2020

tallclair reviewed May 6, 2020

View reviewed changes

dims force-pushed the new-go-runner-image-for-distroless branch from 67f0a5f to 1abc489 Compare May 6, 2020 21:41

BenTheElder reviewed May 7, 2020

View reviewed changes

build/go-runner/go-runner.go Outdated Show resolved Hide resolved

BenTheElder reviewed May 7, 2020

View reviewed changes

build/go-runner/go-runner.go Outdated Show resolved Hide resolved

dims force-pushed the new-go-runner-image-for-distroless branch from f20cab5 to 4eece4e Compare May 7, 2020 01:27

New go-runner image for distroless scenarios

393e095

Signed-off-by: Davanum Srinivas <davanum@gmail.com>

dims force-pushed the new-go-runner-image-for-distroless branch from 4eece4e to 393e095 Compare May 7, 2020 01:29

k8s-ci-robot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. lgtm "Looks good to me", indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels May 7, 2020

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 7, 2020

k8s-ci-robot merged commit 9f76518 into kubernetes:master May 7, 2020

k8s-ci-robot added this to the v1.19 milestone May 7, 2020

dims mentioned this pull request May 7, 2020

Promote the new go-runner image - v0.1.0 kubernetes/k8s.io#849

Merged

tallclair mentioned this pull request May 8, 2020

Rebase most docker images on gcr.io/distroless/static #70249

Closed

10 tasks

listx mentioned this pull request May 8, 2020

Revert "Fix SHA for go-runner manifest - bump version" kubernetes/k8s.io#855

Merged

This was referenced May 10, 2020

Update base image exceptions list kubernetes/sig-release#1084

Merged

Release blocking issues (1.19.0-beta.0, 1.18.3, 1.17.6, 1.16.10) kubernetes/release#1281

Closed

serathius mentioned this pull request Jun 22, 2021

Json format should support same set of feature flags as klog #99270

Closed

pohly mentioned this pull request Nov 3, 2021

package log runner as part of Kubernetes releases #106123

Merged

Elimuhub-coder approved these changes Aug 22, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New go-runner image for distroless scenarios #90804

New go-runner image for distroless scenarios #90804

dims commented May 6, 2020 •

edited

dims commented May 6, 2020 •

edited

dims commented May 6, 2020

dims commented May 6, 2020

dims commented May 6, 2020

dims commented May 6, 2020

justaugustus commented May 6, 2020

tallclair left a comment

BenTheElder commented May 7, 2020 •

edited

BenTheElder commented May 7, 2020

BenTheElder commented May 7, 2020

dims commented May 7, 2020

dims commented May 7, 2020

dims commented May 7, 2020

tallclair commented May 7, 2020

dims commented May 8, 2020

BenTheElder commented May 8, 2020 via email

k8s-ci-robot commented Aug 22, 2022

New go-runner image for distroless scenarios #90804

New go-runner image for distroless scenarios #90804

Conversation

dims commented May 6, 2020 • edited

dims commented May 6, 2020 • edited

dims commented May 6, 2020

dims commented May 6, 2020

dims commented May 6, 2020

dims commented May 6, 2020

justaugustus commented May 6, 2020

tallclair left a comment

Choose a reason for hiding this comment

BenTheElder commented May 7, 2020 • edited

BenTheElder commented May 7, 2020

BenTheElder commented May 7, 2020

dims commented May 7, 2020

dims commented May 7, 2020

dims commented May 7, 2020

tallclair commented May 7, 2020

dims commented May 8, 2020

BenTheElder commented May 8, 2020 via email

k8s-ci-robot commented Aug 22, 2022

dims commented May 6, 2020 •

edited

dims commented May 6, 2020 •

edited

BenTheElder commented May 7, 2020 •

edited